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ABSTRACT 



This document is an outline of a conference presentation on 
developing and implementing authentic oral assessment instruments. Among the 
topics covered are the following: alternative, authentic, direct, 
descriptive, portfolio, and performance assessments. A detailed comparison 
and contrasting is undertaken of alternative and performance assessments. 

Also included is an overview of assessment instruments and the relationship 
between assessment instruments and various characteristics of alternative and 
performance assessments specifically covering business simulations, 
presentation checklists, and oral interviews. The presentation concludes with 
a series of questions to consider when developing an assessment instrument, 
covering such areas as feasibility, the creation process, implementation, and 
interpretation. (Contains 18 references.) (KFT) 
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Background: 




Alternative Assessment 



& 



Performance Assessment 



Similarities: 

• Require students to perform, create, or produce something (a constructed response) 

• Elicit language samples as close to real life as possible (making them authentic) 

• Assess the actual performance of relevant tasks (not mere knowledge) 

• Integrate multiple language skills and sometimes language skill with content knowledge 

• Usually consider both the process and the product 

• Provide in-depth information about a student's skill or mastery 

• Often require higher order thinking skills 

Differences: 



Alternative Assessment 



Performance Assessment 



Tends to be smaller scale 

Extends regularly used and meaningful 

instructional activities 

Sometimes uses rubrics 

Less concerned with validity & reliability 



Tends to be larger scale 
Often independent of instructional 
activities; created on the basis of a needs 
analysis 

Almost always utilizes rubrics 
More concerned with validity and 
reliability 



sue 

U/iniiiirnifiBiai 



All content not otherwise cited is © 1999,2000 Jennifer Maden and/or Michael Taylor. Permission is granted for use and 
adaptation by individual instructors in instructional situations of all material copyrighted by them. All other copyrighted 
material has all rights reserved. 



3 



TESOL 2001, Developing and Implementing Authentic Oral Assessment Instruments, 

Jennifer Maden & Michael Taylor, p. 2 



Deflnitions: 

Alternative Assessment Performance Assessment 



Refers to procedures and techniques that 
can be used within the context of instruction 
and can be easily incorporated into the daily 
activities of the school or classroom. It 
consists of any method of finding out what a 
student does know or can do and is intended 
to show growth and inform instruction and is 

an alternative to traditional forms of testing. 
fStiggins, 1991, cited in O'malley & Valdez-Pierce, 
1995, p. 1 and Navarette et ai cited in Hamayan, 
1995, p. 213) 


Requires students to accomplish complex 
and significant authentic tasks that are 
approximations of real-life while bringing to 
bear prior knowledge, recent learning, and 
relevant skills in the process of doing the 
task. 

(Brown & Hudson, 1998, p. 662; Herman, Aschbacher 
<Sc Winters, 1992, p. 2 qtd. in O'Malley & Valdez- 
Fierce, 1995, pp. 4, 5) 


Alternative Assessment 


Performance Assessment 


• Checklists 


• Interviews 


• Videotapes 


• Role Playing/Simulations 


• Audiotapes 


• Presentations 


• Self-Evaluation/Self Assessment 


• Oral Reports 


• Teacher Observations 


• Group Discussions 


• Interviews with teachers 


• Problem-Solving Tasks 



Advantages: 

• Provides more meaningful feedback to students than a traditional test score 

• Gives stakeholders a deeper insight into students' language proficiency than traditional tests 

• Reflects a wide range of abilities and skills in language in a variety of contexts 

• Elicits communication skills and strategies similar to those needed in real-life situations. 

• Addresses various learning styles 

• Addresses various levels of learning (e.g.. Bloom’s taxonomy of knowing, comprehending, applying, 
analyzing, synthesizing, and evaluating). 

• Can provide a powerfiil, positive washback effect on curriculums with strong conummicative performance 
objectives 

Disadvantages: 

• Reliability may be limited by rater inconsistencies, limited number of observations, and subjectivity in the 
scoring process. 

• Validity may be problematic due to inadequate sampling of the skills being assessed, the sensitivity of 
alternative assessment to test method, task type, and scoring criteria, and variance in performance due to 
factors independent of those being assessed. 



Background information compiled from the following sources: Brown & Hudson ( 1 998), Clapham (2000), Hamayan (1995), Huerta- 
Maclas (1995), O'Malley & Valdez Pierce (1996), and Shohamy (1995). 
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Overview of Assessment Instruments; 

Things Common To All Instruments: 

• Created to assess specific instructional objectives of the course 

• Rubric or checklist created to evaluate performance 

• Related activity intended to elicit authentic language sample reflective of a real life situation the 
learners are likely to encounter on a regular basis 



Differences Between Instruments: 

Instrument 1: Instrument 2: 

(Business Simulation) (Presentation) 


Instrument 3: 
(Oral Interview) 


• Rubric was created before • Checklist was created & • 

students made their continually revised 

performance. throughout the course. 


Rubric was created after 
students had completed their 
interview. 


• Designed over a significant • Small daily incremental • 

period of time. adjustments were made after 

an initial baseline was 
estabUshed. 


Designed within a short 
amount of time. 


• Designed as a final • Designed to address course • 

assessment tool to be used at and students’ own objectives 

the end of the course. and goals. 


Designed within the ongoing 
nature of a course to measure 
intermediate and final 
objectives. 


• Designed as a team effort by • Designed collaboratively • 

two colleagues. between teacher and students. 


Designed by an individual 
teacher. 


• Performances were video • Performances were sometimes • 

taped for student review. videotaped for self and peer- 

evaluation with teacher 
support. 


Performances were 
audiotaped. 


• Final evaluation is a holistic • Final evaluation is a completed • 

score in relation to a band checklist with rankings from 

defined by specific “very good” to “needs work” 

requirements, skills, and on various aspects, sub-skills, 

qualities evidenced in the and best practices for the task 

simulation. required. 


Final evaluation is comments 
followed by a percentage 
grade based on a minimal 
rubric with numerical points 
weighted according to the 
required tasks, topics, and 
objectives for the course and 
specific interview. 


• Students receive a simplified • Students receive the completed • 
written evaluation form. checklist from teacher, may 

also complete a self-evaluation, 
and/or peer evaluations. 


Students receive a copy of the 
teacher’s evaluation form. 


ERIC 5 
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Relationship Between Assessment Instruments and Various Characteristics of 
Alternative & Performance Assessment; 



Characteristics of Alternative Assessment Instruments Assessment 

Instruments 





1 


2 


3 


1 . Requires students to perform, create, produce or do something 


• 


• 


• 


2. Uses real-world contexts or simulations 


• 


• 


• 


3. Extends regularly used and meaninghil instructional activities 


• 


• 


• 


4. Focuses on processes as well as products 


+/- 


• 




5- Taps into higher level thinking and problem-solving skills 


• 


• 




6. Provides information about both the strengths and weaknesses of students 


• 


• 


• 


7. Views language holistically (language is assessed not so much as a structure, but as a tool for 


• 


• 


+/- 


communication and self-expression) 








8. The tasks used in assessment are valued in the real world by the students 


o 


• 


• 


9. Ensures that people, not machines, do the scoring, using human judgment 


• 


• 


• 


10. Encourages open disclosure of standards and rating criteria 


• 


• 


• 


1 1 . Calls upon teachers to perform new instructional and assessment roles (as compared to traditional 


• 


• 


• 


testing) 








12. Results can be used to improve instruction based on accurate knowledge of student progress 


+A 


• 


• 


13. Reflects actual language use and performance (activities/tasks have an authentic communicative 


• 


• 


• 


function; tend to be based on performance in authentic situations likely to encountered in daily life) 








14. Views learning integratively 


• 


• 


• 


15. Appropriate to individual learners' needs (reveals information about a learner's proficiency in the 


• 


• 


• 


context of what is relevant to that learner's life and experiences) 









Churacteristics df P&formitnci Assessment Ihstrutnerits 


Assessment 
Instruments 
1 2 3 


1. Constructed Response: students construct a response, provide an expanded response, engage in a 
performance^ or create a product 


• 


• 


• 


2. Higher-Order Thinking: in responding to open ended questions, the student typically encounters 
complex thought processes and multiple responses/solutions are potentially correct 


• 


• 




3. Authenticity: tasks are meaningful, challenging, and engaging activities that mirror good 
instruction or other real-world context where the student is expected to perform 


• 


• 


• 


4. Process and Product: procedures and strategies for deriving the correct response or for exploring 
multiple solutions to complex tasks are often assessed as well (as or sometimes instead of) the 
product or the "correct" answer 


+A 


• 




5. Depth versus Breadth: performance assessments provide information in depth about a student's 
skil ls or mastery as contrasted with the breadth of coverage more typical of multiple-choice tests 


• 


• 


• 


6. Integrative: the tasks call for integration of language skills and, in some cases, for integration of 
knowledge and skills across content areas 


• 


• 


+/- 


7. Almost Always Has Rubrics or Checklists 


• 


• 


• 



Above adapted from Brown & Hudson (1998, p. 654-5), Clapham (2000, p. 152), 
Shohamy (1995, p. 191), and OMalley & Valdez-Pierce (1995, p. 5). 
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Assessment Instrument #1 ; Business Simulation Instrument 

Summary of Simulation: 

As part of the course, students work for nine weeks in "Corporate Management Teams" developing a number of 
documents and holding simulated meetings relative to an hypothetical company that they create. The final presentation 
requires them to synthesize all they have done and present their hypothetical company to a group of venture capitalists in an 
attempt to persuade them to give the capital necessary to get the company off the ground. The presentation lasts fifteen 
minutes and is followed by a fifteen-minute question and answer session. Those not presenting play the role of the venture 
capitalists. 

Relevant Course Objectives: 

The students will be able to: 

• Use communication strategies introduced or reviewed in class and apply them in role-play settings as well as simple 
authentic business contexts. 

• Prepare for and give presentations in front of the class, giving particular consideration to content (choosing an appropriate 
topic and organizing the information) and delivery (eye contact, projection, speed, use of visual aids, and body language). 

• Use appropriate grammatical structures to communicate ideas clearly during oral and written communication activities. 

Significance of Instrument: 

• Requires students to synthesize and integrate many aspects of their term-long work in "Corporate Management Teams": 

• development of a corporate mission statement for their simulated company 

• creation of simple documents, graphics, and visual aids typical of those used in authentic business 
meetings 

• familiarization with and practice of negotiation processes and skills necessary to successfully function 
during formal business meetings 

• shared insights into why real companies and entrepreneurs highlighted in their textbook have 
succeeded or failed 

• past performances in oral presentations and business simulation role plays 

• Requires students to integrate all of the linguistic skills they have developed and practiced during the course including: 

• pragmatic and sociolinguistic skills (e.g., for giving presentations and for negotiating) 

• lexical, syntactic, grammatical , and phonological skills 

• listening and speaking skills 

• Simulates an unpredictable and challenging real-world situation which is similar in form and function, even if not 
necessarily in all details, to one that they are likely to encounter in real life. 

Overview of Implementation: 

• Several weeks prior to the completion and performance of the presentation, class discussions begin concerning the final 
project. A 2-page prompt is distributed to students to supplement and reinforce the class discussions. 

• During the two weeks between prompt distribution and performance, portions of class time are allocated to allow 
students to begin to work in their groups, pool information, and divide responsibilities. 

• During the week before the performance, a workshop class is held where students practice and get feedback on 
formulating appropriate questions for the question and answer segment of each presentation 

• During the performance week, one group performs per day and their performance is videotaped for their review (note 
that the teacher does not review the video). 

Overview of Prompt: 

The prompt included the following information: 

• A review of previous class activities relevant to this final presentation and how this presentation fit into and flowed out 
of everything students had been doing so far in the course 

• A clear mission statement — what this presentation was supposed to accomplish and how it was to be conducted 

• Reference to an attached set of criteria by which students* final presentations would be graded 

• Advice on the best way to approach preparing for the presentation and question and answer session 

• Details of what would happen in and out of class in the weeks before the presentation to help students prepare for the 
presentation 



Course Objectives 0 1997, Drexel University, English Language Center. All rights reserved. 

Prompt and Rubric © 1999, JenniferMaden and Michael Taylor. Permission granted for use and adaptation by individual instructors in 
O nstructional situations. 
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Assessment Instrument#!: Presentation Checklist 

Presentations - Evaluation Form (Initial Version) 

Presenter: Topic: Date: 

Scoring: Very Good = + (plus) Acceptable = ^ (check mark) Needs Work = - (minus) Not Applicable = N/A 



+ 

+ 

+ 

+ 



+ 

+ 

+ 



+ 

+ 

+ 

+ 



+ 

+ 

+ 



Introduction 



✓ 


- N/A 


Greeting the audience 


✓ 


- N/A 


Statement of credibility 


✓ 


- N/A 


Statement of purpose 


✓ 


- N/A 


Sign-posting for organization of body 






Body 


✓ 


- N/A 


Content is appropriate for purpose and/or audience 


✓ 


- N/A 


Topic is well organized 


✓ 


- N/A 


Sign-posting appropriately used throughout body 






Conclusion 


✓ 


- N/A 


Summary of key points 


✓ 


- N/A 


Distributes support dociunentation 


✓ 


- N/A 


Closing formalities 


✓ 


- N/A 


Inviting questions 






Nonverbal Components 


✓ 


- N/A 


Body language 


✓ 


- N/A 


Eye contact 


✓ 


- N/A 


Voice 



Comments: 



O 



P 2000, Jennifer Maden. Permission granted for use and adaptation by individual instructors in instructional situations. 
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Presenter: 



Presentations - Evaluation Form (Final Version) 

Topic: Date: 



Scoring: Very Good = + (plus) Acceptable = ^ (check mark) Needs Work = - (minus) Not Applicable = N/A 



L Content 


+ 


✓ 


- N/A 


+ 


✓ 


- N/A 




✓ 


- N/A 


+ 


✓ 


- N/A 


+ 


✓ 


- N/A 


+ 


✓ 


- N/A 




✓ 


- N/A 


+ 


✓ 


- N/A 


n 


• Organizat 


+ 


✓ 


- N/A 


+ 


✓ 


- N/A 


+ 


✓ 


- N/A 


+ 


✓ 


- N/A 


+ 


✓ 


- N/A 


+ 


✓ 


- N/A 


+ 


✓ 


- N/A 


+ 


✓ 


- N/A 


+ 


✓ 


- N/A 


+ 


✓ 


- N/A 


+ 


✓ 


- N/A 


+ 


✓ 


- N/A 




✓ 


- N/A 


+ 


✓ 


- N/A 


+ 


✓ 


- N/A 


+ 


✓ 


- N/A 


+ 




- N/A 


+ 


✓ 


- N/A 


+ 


✓ 


. N/A 


+ 


✓ 


- N/A 




✓ 


- N/A 


m. 


DeKvery 


+ 




- N/A 


+ 


✓ 


- N/A 


+ 


✓ 


- N/A 


+ 


✓ 


- N/A 




✓ 


- N/A 


+ 


✓ 


- N/A 




✓ 


- N/A 


-f 


✓ 


- N/A 


+ 


✓ 


- N/A 


+ 


✓ 


- N/A 


+ 


✓ 


- N/A 


+ 


✓ 


- N/A 


+ 


✓ 


- N/A 


+ 


✓ 


. N/A 


+ 


✓ 


- N/A 




✓ 


- N/A 


+ 


✓ 


- N/A 


+ 


✓ 


. N/A 




✓ 


. N/A 



The speaker was well prepared and knowledgeable about the topic. 

The speaker’s goals were apparent (i.e., the purpose was clear). 

The speaker presented the material in a logical manner. 

The speaker remained on target and focused on the appropriate topic. 

The speaker defined key terms and concepts. 

The speaker used specific facts (i.e., a form of support material). 

The speaker used opinions for support (i.e., a form of support material). 

The speaker appropriately used visual aids such as charts, graphs, outlines, handouts, etc. 



Introduction 

The speaker grabbed my attention at the beginning. 

The speaker clearly stated who he/she is and why he/she was speaking to the group. 
The speaker related him/herself to the audience members. 

The opening remarks made me want to listen to the rest of the presentations. 

The speaker clearly identified the purpose of the presentation. 

The speaker actively involved and engaged the audience. 

The speaker prepared me for the body of the presentation. 

The speaker used appropriate sign posting throughout the introduction. 

The speaker appropriately addressed “audience question control.” 

Body 

The central idea and main points were clear and understandable. 

The speaker provided support (examples, statistics,. etc.). 

The presentation was well organized and easy to follow. 

The speaker used technical language and jargon accurately and appropriately. 

The speaker’s reasoning seemed fair and logical. 

The speaker made smooth transitions from one idea to another (i.e., signposting). 

Conclusion 

There was a sense of con:q3letion at the end of the presentation. 

The speaker recapped, restated, summarized, and/or reinforced the main idea/point. 
The speaker used appropriate sign posting throughout the conclusion. 

It is clear for the audience what message the speaker wanted them to take away. 

The speaker managed to create an impact with the ending statements. 

The speaker thanked the audience. 



Vocal Components 

Vocal pitch and intonation (optimum quality/high/low/monotone). 

Vocal intensity (optimum volume/too loud/too soft). 

Vocal variety (pitch, volume, rate). 

Vocalized pauses (urns, ers, okay, like, etc.) 

Rate of speech (speed/pacing). 

Pronunciation (comprehensible/not comprehensible). 

Nonverbal Components 

General appearance. 

Facial expressions. 

Eye contact. 

Gestures. 

Sincerity. 

Credibility. 

Passion, energy level, enthusiasm, and/or intensity. 

Overall impression of confidence in self 

Relation to Audience 

Audience awareness (clear understanding of what they understood, need, want, etc.) 
Content and topic appropriate to audience. 

Engaged, empathized with, and/or interested audience. 

Responded well to interruptions and questions. 

Remained in control of audience. 



2000, Jennifer Maden. Permission granted for use and adaptation by individual instructors in instructional situations. 
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Assessment Instrument #3: Oral Interview 



Relevant Objectives Derived from Curriculum Guidelines, Textbook, and Class Activities: 

By the end of this elementary level speaking and listening class, students will be able to: 

• Ask and respond to questions about their immediate environment and situations and topics of personal interest. 

• Develop vocabulary to talk about their family and home, personal interests, and objects in their immediate environment. 

• Participate in daily conversations in a limited way. 

• Produce basic grammatical structures including simple past and present tenses. 

• Describe other people, talk about changes, and talk about and future plans. 



Student Name: Date: 

Tell me about your family. 

• tells about brothers, sisters, and parents (5 pts.) 

• gives some information about them (ex; what they do, where they live) (5 pts.) 

• produces a good quantity of speech (3 pts.) 

• speech is connected (not many pauses) (4 pts.) 

• accuracy (3 pts.) 

Comments: 


(Points Received) out of \ 
Total Points Possible 


out of 20 total 

out of 5 

out of 5 

out of 3 

out of 4 

out of 3 


How often do you exercise? Do you play any sports? How good are you at sports? 

* Answers each question (2 pts. each) 

• Grammatical accuracy (1 pt. per question) 


out of 9 total 

out of 2 

out of I 


What did you do last weekend? 

• Tells what he/she did (3 pts.) 

• Elaborates ^propriately on what he/she did (3 pts.) 

• Uses Past Tense (2 pts.) 

Comments: 


out of 8 total 

out of 3 

out of 3 

out of 2 


Fm looking for Paul. Can you tell me what he looks like (show a picture)? 

• Uses at least 4 descriptive adjectives (non-clothing) (4 pts.) 

« Desoibes clothing of the person (2 pts.) 

9 Grammar (1 pt) 

• Pronunciation (2 pts.) 


out of 9 total 

out of 4 

out of 2 

__ out of 1 
out of 2 


WhaFs your favorite kind of food? Why? Your least favorite kind of food? Why? 

• Gives favorite kind of food (2 pts.) 

• Chves least favorite kind of food (2 pts.) 


out of 4 total 

out of 2 

out of 2 


What are 3 ways you have changed since 1998? 

• Chves at least 3 ways has changed (6 pts.) 

• Uses Correct Grammar (2 pts.) 


out of 8 total 

out of 6 

out of 2 


What are 3 things you are going to do after you finish studying English? 

• Gives at least 3 things he/she is going to do (6 pts.) 

• Uses Correct Grammar (2 pts.) 


out of 8 total 

out of 6 

out of 2 


Overall Comments: 


Final Grade 

/66 pts. = 
% 
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Questions to CoNsroER in Developing your own Assessment Instruments; 



Feasibility 

1 . Is your proposed instrument small enough for you to be able to administer it independently of others? If not, 
will you have the support and cooperation of the additional parties that you will need? 

2. How much class time is available for individual vs. group performances? 

3. How much time is necessary for creation of the assessment instrument, rubric, or checklist, and for giving 
feedback to students? {Remember the range presented. "Small" instruments can still be valuable. Instrument 
3 was created in an hour!) 

4. Is this assessment time and/or cost-effective? 

5 . When will the rubric/checklist be created - before, during, or after the activity to be evaluated? (Remember 
again that although "after" is not preferable, it is often better than not at all) 

6. \^o will be involved in generating Ae criteria to be assessed - an individual teacher, a team of teachers, 
students in collaboration with teachers? 



The Creation Process 

7. Are there easting instruments available that can be adapted? 

8. Do the activities and tasks selected represent the content and sk ill s that students are expected to master? 

9. Do the activities involved allow students to demonstrate their skills and abilities? 

10. Do the assessment results address the goals and objectives of the curriculum, course, or program? 

11. Does the assessment cover topics that are immediately useful to the learners? 

12. For uppra" level students, does this assessment require a task that is cognitively and/or linguistically complex? 

13. Does this assessment cover an adequate range of content? 

14. Is this assessment fair? 

15. Is this assessment meaningful? 

16. What is the proportion of "language" vs. "domain knowledge" to be assessed? 

17. What relative weighting should be given to the different criteria? 



Implementation and Interpretation 

18. How can the evaluation criteria reflect the kinds of judgments and consequences that a successful (and then 
less successful) performance would entail? 

19. How can the scoring information be interpreted and presented in ordra" to give maximum information back to 
the test takers and to the teacher? 

20. Can the results inform me as a teacher eithw; how to respond to current learners’ needs for the remainder of 
the class; or to future learners’ needs when revisiting this task with a new group of learners? 

2 1 . Do you feel competent to judge this kind of task? If not, can you find someone to judge the aspects you feel 
uncomfortable judging? Would students and the administration respect that person's judgment? 



Above adapted in part from Hamayan(l 995, p. 219), O’Malley & Valdez Pierce (1996, pp. 27-31), Pool & Bracey (1992, pp. 40,41), and 

Shohamy (1995, p.l91), as well as from die presenters ' experiences. 
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